Optimize truncateFileTree token estimation and sampling seed - #1216

Nice diagnosis on the seed and countTokensJson issues — using JSON.stringify(sortedFiles) purely as a random seed was wasteful, and if printedTree is genuinely consumed as a raw string downstream (not JSON-encoded), switching to countTokens is a legitimate accuracy + perf win. The dir/extension list split (UNIMPORTANT_DIR_PATTERNS / UNIMPORTANT_EXTENSIONS) is a clean way to avoid the startsWith('/') check per file per extension.

However, in the extension list refactor at the bottom of truncate-file-tree.ts, .so disappears entirely — it's present in the old unimportantExtensions array (grouped with .exe, .dll, .lib) but missing from the new UNIMPORTANT_EXTENSIONS. That's an unannounced behavior change: shared libraries will now show up in the file tree where they were previously filtered. If intentional, call it out in the PR description; if accidental (looks like it), please restore it and add a regression test — your new test suite covers .min.js filtering but nothing for the executable/library category, so this slipped through untested.

Separately, the countTokensJson → countTokens swap changes token-budget behavior across the whole truncation pipeline (four call sites). That's a bigger functional change than the PR title suggests ('optimize... estimation') — it's really a correctness change to what gets counted. Worth confirming with whoever owns token-counter.ts that printedTree was never intended to model JSON-escaped cost, since if it was, this now under-counts tokens and could cause budget overruns in production.

Good test additions overall; would like one covering the extension-list edge case before this goes in.

Thanks for the keen eye @codebuff-team! Both points have been addressed in the latest commit:

1. Restored .so Extension & Added Regression Test:

   • The omission of .so from UNIMPORTANT_EXTENSIONS was indeed accidental during the list split. It has been restored alongside .exe, .dll, and .lib.
   • Added a regression test in truncate-file-tree.test.ts verifying that compiled binary and library files (.so, .dll, .exe, .lib) are filtered out by removeUnimportantFiles.

2. countTokens vs countTokensJson Rationale:

   • Verified the downstream usage in packages/agent-runtime/src/system-prompt/prompts.ts: printedTree is directly interpolated into raw markdown template literals:

     <file_tree>
     ${printedTree}
     </file_tree>

     It is never serialized as JSON.
   • Calling countTokensJson was running JSON.stringify(string) under the hood, which wrapped the string in double quotes and escaped every newline (\n -> \\n), artificially inflating token estimates and spending time copying strings. countTokens(printedTree) reflects the true token count of the raw string sent to the model.